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Overview 


•  ACCM  Background 

•  Test  Details 

•  Data  Collection 

•  Test  Analysis  Methodology 

•  Findings 

•  Lessons  Learned  and  Conclusion 


ACCM  Background 


•  ACCM  is  a  Warfighter 
Rapid  Acquisition  Program 
(WRAP)  involving 
AFSOC/A5T,  AFRL/DE, 
AFRL/HE,  and  Boeing 
Scorpworks  Lab 

•  Laser  system  designed  to 
provide  significant  glare 
source 


Test  Details 


•  Main  purpose;  to  determine  if  the  ACCM  laser  system 

works  as  an  effective  counter  measure  against  small 

arms  fire 

•  Three  test  phases 

-  No  laser  (no  beam) 

-  Low  power  level 

-  High  power  level 

•  Players 

-  Helicopter  gunner 

-  Shooters 


Proposed  Data  Collection  Tools 


•  Multiple  Integrated  Laser  Engagement  System  2000 
(MILES  2000)  gear 

•  Video  feeds 

•  Shot  placement  software 

•  Sensor  suite 

-  Accelerometer  (rifle  recoil),  optical  (MILES/ ACCM 
beam),  data  logger  (GPS  position,  time  etc.) 

•  Shooter  Data 

-  Interviews  and  surveys 

•  Gunner  Data 


HE  Test  Methodology 


•  Measures  Of  Performance  (MOPs)  considered 

•  Comparing  test  phases 

•  What  we  wanted  to  do  with  our  data 

•  What  we  were  able  to  do  with  our  data 


MOPs  Considered 
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•  Hit  ratio  on  the  helo 

-  No.  of  hits  divided  by  shots  fired 

-  A  hit  was  designated  a  shot  within  1 1  ’  of  the  center  of 
the  gunner’s  window 

•  Average  miss  distance  and  Circnlar  Error  Probable 
(CEP) 

•  Average  number  of  aggressors  killed 

•  Average  number  of  near-misses 


Comparing  Test  Phases 


•  Compare  the  MOPs  captured  via  statistical  tests 

-  Large  sample  hypothesis  tests 

-  Determine  if  shooters  performance  was  adversely 
affected  in  engagements  with  the  ACCM  laser  system 

•  Analyze  survey  responses 

-  Assigned  a  score  to  each  response  and  looked  at 
averages  and  standard  deviation 

-  Did  not  look  at  non-parametric  statistics 


Data  -  Hopes  vs.  Reality 


•  Hopes 

-  Analyze  each  shooter’s  performance  individually 

•  Shooter  variability  not  an  issue 

-  Shot  placement  software  would  efficiently  “score”  the 
shots 

•  Reality 

-  Without  sensor  suite  could  not  analyze  the  shooter’s 
performance  individually  (assume  ea.  shooter  the  same) 

-  Without  shot  placement  software  all  of  the  videos  had 
to  be  watched  and  scored  by  “hand” 


Findings 


How  do  you  conduct  meaningful  analysis  based  on  only  42%  of 

the  data  points? 


No  Beam 

0.5%  MPE 

1%  MPE 

Total 

Total  Fired 

3217 

2162 

3034 

8413 

Total  Found 

1406 

859 

1272 

3537 

%  Found 

0.4371 

0.3973 

0.4192 

0.4204 

•  Only  found  42%  of  the  shots 

-  Remaining  shots  were  either  not  seen/captured  on  the  video  feeds 
or  missed  the  hangar  all  together 

-  Non-representative  sample 

•  Most  of  MOPs  could  not  be  used 

-  Except  for  hit-ratio,  kills,  and  near-misses 


Y  Coordinates  (ft) 


Shooter  Accuracy 


Low  Power  Hits 


Shooter  Accuracy 


No  Beam 

Low  Pwr 

High  Pwr 

Total  Hits 

314 

117 

274 

Shots  Fired 

3217 

2162 

3034 

Hit  Ratio 

0.0976 

0.0541 

0.0903 

•  Hit  ratio  is  statistically  smaller  in  the  Low  Power  test 
phase 

•  Looked  into  this  further  by  analyzing  hit  ratio  at  the 
engagement  level 

-  No.  of  hits  per  engagement 

-  No.  of  engagements  with  5, 10, 15,  or  20+  hits 

-  Analyzed  this  for  all  of  the  engagements  and  a  random 
sampling  of  engagements 

•  Consistent  results 


Findings 


•  One  other  factor  changed  with  the  power  of  the  laser 
(which  we  were  not  made  aware  of  until  late  into  the 
analysis) 

-  Spot  size  went  from  29.5’  in  diameter  in  High  Power  test 
phase  to  42.7’  in  diameter  in  the  Low  Power  test  phase 

-  A  difference  of  744  square  feet  (or  double  the  area) 

•  So  we  conclude  that  the  laser’s  spot  size  is  the  most 
important  factor,  but  more  testing  needs  to  be  done  to 
confirm  this 


Findings 
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•  Shooters  killed  and  near-misses  by  gunner 

-  A  lot  more  kills  and  near-misses  from  the  No  Beam  to 
the  High  Power  test  phase 

-  Explanation:  Gunner’s  are  used  to  aiming  using  tracer 
rounds,  cannot  do  that  when  using  blanks 

•  Laser  became  their  aiming  device 

•  Overall  our  findings  were  not  inherently  conclusive 

-  Missing  a  lot  of  data 

-  Need  data  on  each  shooter’s  performance 

-  Better  way  to  score/find  the  shooter’s  shots 


Lessons  Learned 


•  Test  environment  is  ever  changing 

-  Flexibility 

-  Back-up  plans 

•  Understand  all  of  the  possible  variables/factors  prior 
to  test 

-  Control  as  many  as  possible 

•  Everything  sonnds  great  on  paper  (but  chances  are 
things  will  not  work  as  advertised) 

•  More  testing  to  obtain  conclusive  results  is  never  a 
conclusion  that  wants  to  be  heard 


Questions? 


Backups 


ACCM  Background 


•  Main  purpose;  to  determine  if  the  ACCM  laser  system 

works  as  an  effective  counter  measure  against  small 

arms  fire 

•  ACCM  is  a  Warfighter  Rapid  Acquisition  Program 
(WRAP)  involving  AFSOC/A5T,  AFRL/DE, 
AFRL/HE,  Boeing  Scorpworks  Lab,  and  AFMC/OAS 

•  Laser  system  designed  by  Boeing  Scorpworks  lab  to 
provide  significant  glare  source 

-  Green  light  laser  of  particular  wavelength,  found  to 
create  a  ‘dazzling  effect’  on  the  human  eye 

•  Designed  to  fill  weapons  engagement  zone  gap  from 
IKm  to  terminal  area  of  recovery 


Test  Details 


•  Helicopter  gunner 

-  On  scissor  lift  in  hangar  (gunner’s  window) 

-  Goal  was  to  “kill”  as  many  shooters  as  possible  during 
each  engagement 

-  Weapon  was  a  M-249  (equipped  with  MILES  2000) 

•  Shooters  in  the  field  in  front  of  hangar 

-  Two  teams  of  5  shooters 

-  Goal  was  to  get  as  many  shots  on  the  helicopter  as 
possible  (aim  point  -  center  of  the  gunner’s  window) 

-  Weapon  -  M-4  rifles  (equipped  with  MILES  2000) 


Test  Details 


•  Multiple  Integrated  Laser  Engagement  System  2000 
(MILES  2000) 

-  System  of  sensors  and  transmitters  that  the  shooters 
and  gunner  wear 

•  Gunner  did  not  wear  a  sensor  so  we  could  not  determine 
when  he  was  hit  -  did  not  want  his  weapon  to  be  disabled 
during  engagement 

-  Record  hits  and  near-misses  (disables  weapon  if  hit) 

•  Main  purpose:  to  determine  if  the  ACCM  laser  system 
works  as  an  effective  counter  measure  against  small 
arms  fire 
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OAS  Involvement 


•  Independent  review  of  the  Human  Effects  test  for  the 
ACCM  program 

-  OAS  holds  no  stake  in  the  outcome  of  the  WRAP 

•  Test  design,  implementation,  and  analysis  of  results 

-  OAS  was  involved  in  previous  phase  of  HE  test 

•  Production  of  study  report  to  include  findings  and  future 
recommendations 


Data  Collection 


3  cameras  for  video  shot  placement 

IR  sensitive  cameras  pickup  MILES  2000  pulses 

Shot  placement  software  proved  to  be  ineffective 

All  video  had  a  time  stamp  that  was  synchronized  with  all  other 
data  by  GPS  time 

Each  video  was  scanned  by  team  from  Scorpworks  lab  to 
identify  and  assess  time  and  location  of  each  shot 

Scorpworks  sensor  suite 

Data  loggers  were  found,  during  test,  to  be  unreliable 
•  Made  other  sensors  useless 
Voice  recorders  were  used  but  not  analyzed 

Combat  camera  footage  on  field  during  engagements  to 
verify  sequences  of  action 


Data  Collection 


•  MILES  gear  downloads 

•  Shooter  data 

-  Interviewed  shooters  after  each  engagement  to  record 
shots  fired,  misfires,  j  ams  etc. 

-  3  cameras  for  video  shot  placement 

•  Gunner  data 

-  Shots  fired,  etc. 

•  Shooter  surveys 

-  Handed  out  at  end  of  each  phase  per  night 


Hopes  For  Our  Data 


•  Wanted  to  locate  and  measure  the  miss  distance  of  all 
shots  fired  by  the  aggressor  teams 

-  Use  this  data  to  compare  test  phases  or  conditions 

•  Show  from  surveys  whether  or  not  the  aggressors  had 
opinions  about  particular  test  conditions  that  were 
later  verified  through  analysis  of  shot  data 

•  Show  number  of  kills  and  near-misses  against  the 
aggressors 


Reality  of  Our  Data 


•  Without  a  working  Scorpworks  sensor  suite,  we  were 
unable  to  identify  shots  by  shooter  or  show  when  a 
shooter  was  in  the  ACCM  beam 

-  No  way  to  determine  (by  shooter)  if  a  shot  was  better  or 
worse  while  the  shooter  was  in  the  laser’s  path 

•  Without  the  shot  placement  software  all  of  the  videos 
had  to  be  watched  and  the  shots  scored  “by  hand” 

-  Capturing  a  MILES  2000  pulse  on  hangar,  finding  the 
center,  and  then  calculating  the  radial  miss  distance 


Findings 


•  Shooter’s  accuracy 

-  No  notable  difference  between  the  no  beam  and  1% 
MPE  test  phases 

-  Hit  ratios  were  significantly  lower  in  the  0.5%  MPE  test 
phase  than  in  the  other  two 

•  If  laser  had  a  negative  effect  on  shooter  accuracy 
wouldn’t  the  trend  continue  as  the  power  of  the  laser 
went  up  (brighter)? 


Findings 


Not  much  difference  seen,  with  similar  numbers  of  shots  found, 
in  the  No  Beam  and  High  Power  scatter  plots 


Findings 


•  In  the  Low  Power  condition  we  had  significantly  fewer  data 
points  to  work  with  than  in  the  No  Beam  or  High  Power 
conditions 


